Skip to content

[CORE-16608] cluster_link: validate source SR reachability and target emptiness on link creation#31011

Open
mnajda-redpanda wants to merge 5 commits into
redpanda-data:devfrom
mnajda-redpanda:mnajda/pre-link-creation-validation-checks
Open

[CORE-16608] cluster_link: validate source SR reachability and target emptiness on link creation#31011
mnajda-redpanda wants to merge 5 commits into
redpanda-data:devfrom
mnajda-redpanda:mnajda/pre-link-creation-validation-checks

Conversation

@mnajda-redpanda

@mnajda-redpanda mnajda-redpanda commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Extends the shadow-link preflight (manager::link_preflight_checks, which
already probes the remote Kafka brokers) with two Schema Registry API-sync
checks. They run on the same path, so they gate both the validate_only dry
run and a real create:

  1. Source reachability — build an SR REST client from the link's URL, TLS
    settings, and credentials and list against it (link_sr_unreachable).
  2. Target emptiness — every destination context the link would import into
    must be empty, scoped to the mapped contexts (link_sr_target_not_empty).

Both surface to the admin ConnectRPC as FAILED_PRECONDITION.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v26.1.x
  • v25.3.x
  • v25.2.x

Release Notes

  • none

@mnajda-redpanda mnajda-redpanda changed the title cluster_link: validate source SR reachability and target emptiness on link creation [CORE-16608] cluster_link: validate source SR reachability and target emptiness on link creation Jul 3, 2026
@mnajda-redpanda mnajda-redpanda force-pushed the mnajda/pre-link-creation-validation-checks branch 2 times, most recently from ff5d59a to c6cad96 Compare July 3, 2026 07:47
@mnajda-redpanda mnajda-redpanda self-assigned this Jul 3, 2026
@mnajda-redpanda mnajda-redpanda marked this pull request as ready for review July 3, 2026 07:51
@mnajda-redpanda mnajda-redpanda requested review from a team, Copilot and pgellert and removed request for a team July 3, 2026 07:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Schema Registry (SR) API-sync preflight validation to cluster-link creation so that both validate_only and real creates are gated on (1) source SR reachability/auth and (2) destination-context emptiness for the contexts that would be imported into. This extends existing link preflight checks (Kafka broker probing) and introduces new SR client construction utilities plus unit/integration coverage.

Changes:

  • Introduces SR preflight checker (sr_preflight_checker) and wires it into cluster_link::manager::link_preflight_checks, surfacing failures as FAILED_PRECONDITION.
  • Adds a shared SR REST client builder (schema_registry_sync::make_source_sr_client) + context-filter helper (filter_selects_source_context) to keep preflight logic consistent with runtime SR behavior.
  • Adds new C++ unit tests + ducktape end-to-end tests for SR preflight scenarios (unreachable source, non-empty target, mapping/filter scoping).

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tools/type-checking/type-check-strictness.json Adds new ducktape test to strict type-check set.
tests/rptest/tests/shadow_link_sr_preflight_test.py E2E ducktape coverage for SR reachability + target emptiness checks on validate/create.
src/v/schema/tests/fake_registry.h Extends fake registry interface with has_subjects.
src/v/schema/tests/fake_registry.cc Implements has_subjects for fake registry store.
src/v/schema/registry.h Adds registry::has_subjects API for efficient emptiness checks.
src/v/schema/registry.cc Implements has_subjects in enabled/disabled schema registry wrappers.
src/v/redpanda/admin/services/shadow_link/shadow_link.cc Updates validate-only path to pass full metadata into test_connection.
src/v/redpanda/admin/services/shadow_link/err.cc Maps new SR preflight errors to FAILED_PRECONDITION.
src/v/pandaproxy/schema_registry/rest_client/BUILD Expands visibility so cluster_link can depend on SR REST client code.
src/v/datalake/tests/record_schema_resolver_test.cc Updates test registry wrapper to implement has_subjects.
src/v/cluster/cluster_link/frontend.cc Reuses shared context-filter predicate from cluster_link::model.
src/v/cluster/BUILD Adds Bazel dep on new sr_context_mapping lib.
src/v/cluster_link/utils.h Exposes TLS config builder used by both Kafka and SR client paths.
src/v/cluster_link/utils.cc Refactors TLS config creation to support SR client construction.
src/v/cluster_link/tests/link_test.cc Adds unit tests for SR preflight behavior and required Kafka API advertisement in mocks.
src/v/cluster_link/tests/deps.h Adds fake_source_sr_prober and test fixture SR registry state.
src/v/cluster_link/tests/deps.cc Wires SR preflight checker into manager fixture construction.
src/v/cluster_link/tests/BUILD Adds deps for SR types + fake registry and additional Kafka protocol APIs.
src/v/cluster_link/sr_preflight_checker.h Defines SR preflight checker and source SR prober interfaces.
src/v/cluster_link/sr_preflight_checker.cc Implements SR reachability probe + destination-context emptiness validation.
src/v/cluster_link/service.h Changes test_connection to accept full model::metadata for richer preflight.
src/v/cluster_link/service.cc Constructs destination schema registry earlier and injects SR preflight checker into manager.
src/v/cluster_link/schema_registry_sync/tests/sr_sync_test_fixtures.h Updates delegating registry fixture to implement has_subjects.
src/v/cluster_link/schema_registry_sync/tests/source_client_test.cc Unit tests for URL/TLS/auth parameter derivation for SR source client.
src/v/cluster_link/schema_registry_sync/tests/BUILD Adds Bazel target for new source_client_test.
src/v/cluster_link/schema_registry_sync/source_client.h Adds SR source client parameter resolution + client factory API.
src/v/cluster_link/schema_registry_sync/source_client.cc Implements SR source URL parsing, TLS/SNI resolution, and client construction.
src/v/cluster_link/schema_registry_sync/BUILD Adds Bazel library target for source_client.
src/v/cluster_link/model/sr_context_mapping.h Declares shared predicate for SR context filter membership.
src/v/cluster_link/model/sr_context_mapping.cc Implements shared predicate used by runtime and preflight.
src/v/cluster_link/model/BUILD Adds Bazel library target for sr_context_mapping.
src/v/cluster_link/manager.h Injects SR preflight checker dependency; updates test_connection signature.
src/v/cluster_link/manager.cc Runs SR preflight after Kafka preflight and updates validate-only logic to use metadata.
src/v/cluster_link/fwd.h Adds forward declaration for sr_preflight_checker.
src/v/cluster_link/errc.h Adds new SR-related error codes.
src/v/cluster_link/errc.cc Adds messages for new SR-related error codes.
src/v/cluster_link/BUILD Adds new sources and deps for SR preflight and SR client construction.

Comment thread src/v/cluster_link/schema_registry_sync/source_client.cc Outdated
Comment thread src/v/cluster_link/schema_registry_sync/source_client.cc Outdated
@mnajda-redpanda mnajda-redpanda force-pushed the mnajda/pre-link-creation-validation-checks branch from c6cad96 to 931d73a Compare July 3, 2026 11:57
@mnajda-redpanda

Copy link
Copy Markdown
Contributor Author

force-pushed to resolve a merge conflict

@bartoszpiekny-redpanda bartoszpiekny-redpanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks great but I'm not fully familiar with the functionality so please wait for someone else accepting it 🙂

@@ -0,0 +1,72 @@
/*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've skipped reviewing this commit, because I think we'll be able to just drop in the source reader client from #30999 instead of this one.

@@ -0,0 +1,30 @@
/*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for splitting this into a separate commit 🚀

Comment on lines +85 to +87
# The API-sync path is feature-gated; ensure it is active so the
# request reaches the preflight checks rather than the feature gate.
self.target_cluster_service.set_feature_active(SR_API_SYNC_FEATURE, True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this bit is necessary here.

The way these features work is that they auto-activate when all of the brokers are at least as up to date as a specific broker version. For this api sync feature specifically, the feature auto-activates when the brokers are all on at least version 26.2.

Since this tests creates a cluster where all brokers are at least 26.2, there is no need to manually enable the feature, it will just be auto-enabled while the cluster starts up.

Comment on lines +106 to +116
// Identity/unset mapping: the in-scope source contexts (those the link's
// filter selects) become the destination contexts. Filter membership is
// resolved through the shared predicate so this stays consistent with the
// runtime client-write blocker (cluster/cluster_link/frontend.cc).
auto identity_targets = [&] {
return source_contexts | std::views::filter([&](const auto& ctx) {
return model::filter_selects_source_context(api.filter, ctx);
})
| std::views::transform([](const auto& ctx) { return ctx(); })
| std::ranges::to<absl::flat_hash_set<ss::sstring>>();
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for identity mapping we need to be slightly stricter than this. We should also validate that there is no context .ctx which doesn't exist in the source + exists in the destination + in scope in the source.

Unless I'm missing something, I think we won't need to consider the contents of the source SR at all, and these emptiness checks will just depend on the state of the config + destination schema registry.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Identity emptiness now reads the destination (list_subject_versions filtered by source scope) and never consults the source. An in-scope context present in the destination but absent from the source is now caught.

} catch (...) {
eptr = std::current_exception();
}
co_await client->shutdown();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this shutdown throw? Wondering if we need to move it into the try-catch above

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — it can. Now wrapped in ss::coroutine::as_future, a failed close is logged and ignored so it can't mask the probe outcome.

@mnajda-redpanda mnajda-redpanda force-pushed the mnajda/pre-link-creation-validation-checks branch from 931d73a to ce6f252 Compare July 6, 2026 15:44
@mnajda-redpanda

mnajda-redpanda commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

force-pushed: review from @pgellert and local Claude incorporated

  • Reachability probe: returns cl_result<void> (was returning contexts); probes with lightweight list_contexts; error handling simplified via as_future.
  • Target emptiness: now derived from destination + config only — catches in-scope destination contexts absent from the source.
  • check() hardening: total (never throws); early is_enabled() guard for disabled target SR; shard-abort → service_shutting_down.
  • Tests: link_test reworked for destination-driven emptiness; ducktape asserts on error messages + seeds target-only.

@vbotbuildovich

vbotbuildovich commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

CI test results

test results on build#86761
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FLAKY(PASS) NodeWiseRecoveryTest test_recovery_local_data_missing {"wait_for_final_manifest_uploads": true} integration https://buildkite.com/redpanda/redpanda/builds/86761#019f382d-fcd7-4c80-b886-c02d19c80642 9/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0650, p0=0.4895, reject_threshold=0.0100. adj_baseline=0.1827, p1=0.4304, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_recovery_local_data_missing
test results on build#86796
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FAIL SchemaRegistrySyncE2ETest test_schema_registry_api_sync_recovers_after_source_unavailable null integration https://buildkite.com/redpanda/redpanda/builds/86796#019f3ca0-65d8-4459-985a-bd68c367bb2c 0/11 Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaRegistrySyncE2ETest&test_method=test_schema_registry_api_sync_recovers_after_source_unavailable
FAIL SchemaRegistrySyncE2ETest test_schema_registry_api_sync_recovers_after_source_unavailable null integration https://buildkite.com/redpanda/redpanda/builds/86796#019f3ca2-e6b0-4ee1-9e28-47f24fed9b4f 0/11 Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaRegistrySyncE2ETest&test_method=test_schema_registry_api_sync_recovers_after_source_unavailable
FLAKY(PASS) ListOffsetsLeaderEpochRedpandaTest test_list_offsets_epoch {"correct_epoch": false} integration https://buildkite.com/redpanda/redpanda/builds/86796#019f3ca2-e6af-4ad1-8362-c8d11be0c5f9 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0014, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ListOffsetsLeaderEpochRedpandaTest&test_method=test_list_offsets_epoch
test results on build#86798
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FAIL SchemaRegistrySyncE2ETest test_schema_registry_api_sync_recovers_after_source_unavailable null integration https://buildkite.com/redpanda/redpanda/builds/86798#019f3ce8-5b62-4943-84db-0375beb42832 0/11 Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaRegistrySyncE2ETest&test_method=test_schema_registry_api_sync_recovers_after_source_unavailable
FAIL SchemaRegistrySyncE2ETest test_schema_registry_api_sync_recovers_after_source_unavailable null integration https://buildkite.com/redpanda/redpanda/builds/86798#019f3ce8-5b65-4c07-8064-6b4d1024662a 0/11 Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=SchemaRegistrySyncE2ETest&test_method=test_schema_registry_api_sync_recovers_after_source_unavailable
test results on build#86801
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FLAKY(PASS) TxAtomicProduceConsumeTest test_basic_tx_consumer_transform_produce {"with_failures": true} integration https://buildkite.com/redpanda/redpanda/builds/86801#019f3d5a-7ec1-4b62-b8e0-b9ed72947796 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0028, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=TxAtomicProduceConsumeTest&test_method=test_basic_tx_consumer_transform_produce

@nguyen-andrew nguyen-andrew left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, will review again once rebased and merged with Gellert's latest stuff

// the destination context and the filter.
ss::future<chunked_vector<ss::sstring>>
identity_offending_contexts(const shadow_schema_registry_api& api) {
auto matches = co_await _destination.list_subject_versions(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do we need all the subject versions here, or would just the subjects suffice? I think in the common case the destination should be empty or have very few subjects/subject-versions, so it probably wouldn't make a difference.

@mnajda-redpanda mnajda-redpanda Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added get_subjects to the schema::registry interface (delegates to the existing sharded_store::get_subjects) and the identity check now lists subjects instead of subject-versions.

Done in af9f868

Comment on lines +102 to +103
# A source subject puts the default context in scope for identity
# mapping, so the check actually queries the (empty) target context.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, your latest push made the emptiness check derive its scope from the link config and only reads the destination registry (and not the source), right? If so, is this comment still accurate?

@mnajda-redpanda mnajda-redpanda Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it's stale - removed it. With the config-scoped, destination-only check the source seed had no effect, so I dropped both the comment and the seeding.

Comment on lines +167 to +168
# A source subject puts the default context in scope for identity
# mapping; the target is left empty so the emptiness check passes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment and seed removed.

@mnajda-redpanda mnajda-redpanda force-pushed the mnajda/pre-link-creation-validation-checks branch from ce6f252 to 7279c68 Compare July 7, 2026 10:34
@mnajda-redpanda

Copy link
Copy Markdown
Contributor Author

force-pushed: applied @nguyen-andrew review comments

  • add get_subjects to the registry interface; identity emptiness check now lists subjects instead of subject-versions
  • drop the now-inert source-subject seeding and stale comments from the two positive preflight tests

Didn't rebase on @pgellert changes yet, will do next.

Adds a short-circuiting emptiness probe forwarding to the existing
sharded_store::has_subjects, plus implementations in every registry
subclass so the interface stays pure-virtual.
Adds a subject listing forwarding to the existing
sharded_store::get_subjects, returning one entry per subject rather
than per (subject, version), plus implementations in every registry
subclass so the interface stays pure-virtual.
Lifts filter_selects_source_context out of frontend.cc into a shared
model target so the runtime client-write blocker and the preflight check
resolve filter membership the same way. Behavior-preserving.
@mnajda-redpanda mnajda-redpanda force-pushed the mnajda/pre-link-creation-validation-checks branch from 7279c68 to 51f1e07 Compare July 7, 2026 12:31
@mnajda-redpanda

Copy link
Copy Markdown
Contributor Author

force-pushed:

  • rebased onto latest dev
  • dropped the source-client commit entirely - used @pgellert implementation, copilot comments are now stale
  • re-bounded the probe with a child abort source fired by a 5s deadline timer - preflight fails fast while the sync path keeps its full 30s (the reader's 30s internal budget outran the admin RPC deadline (→ misleading unavailable) for an unreachable source).

All changes incorporated in be8b85c (didn't include the exact diff because of the force-push, sorry for noisy diff)

@vbotbuildovich

vbotbuildovich commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Retry command for Build#86796

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/cluster_linking_schema_registry_sync_test.py::SchemaRegistrySyncE2ETest.test_schema_registry_api_sync_recovers_after_source_unavailable

@mnajda-redpanda

Copy link
Copy Markdown
Contributor Author

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/cluster_linking_schema_registry_sync_test.py::SchemaRegistrySyncE2ETest.test_schema_registry_api_sync_recovers_after_source_unavailable

@pgellert pgellert self-requested a review July 7, 2026 14:16
@vbotbuildovich

vbotbuildovich commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Retry command for Build#86798

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/cluster_linking_schema_registry_sync_test.py::SchemaRegistrySyncE2ETest.test_schema_registry_api_sync_recovers_after_source_unavailable

Extends manager::link_preflight_checks (already probing the remote Kafka
brokers) with two Schema Registry API-sync checks that gate both the
validate_only dry run and a real create: source reachability
(link_sr_unreachable) and target context emptiness
(link_sr_target_not_empty, scoped to the mapped destination contexts).
Both surface to the admin ConnectRPC as FAILED_PRECONDITION.

Adapt the existing SR-sync recovery E2E test to this behavior:
creating a link against an unreachable source is now rejected up front,
so the test can no longer point a fresh link at a bad URL. It instead
creates the link against the live source, lets the first full sync land,
then severs the destination's egress to the source SR mid-sync
(firewall_blocked) to exercise the unavailable-then-recover path.
Two-cluster coverage of the SR preflight over validate_only and real
create, positive and negative, asserting the specific failure message.
@mnajda-redpanda mnajda-redpanda force-pushed the mnajda/pre-link-creation-validation-checks branch from 51f1e07 to 4dcf8e9 Compare July 7, 2026 16:05
@mnajda-redpanda

Copy link
Copy Markdown
Contributor Author

force-pushed: fix SR-sync recovery E2E test

  • the test used to create a link against a bad URL, which this commit's new preflight now rejects at creation
  • it now blocks the source SR connection mid-sync via firewall_blocked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants